The invention relates to a method and a system for visualizing the environment of a vehicle as well, as to a calibrating device for calibrating the system.
A night vision system for a vehicle is known from German Patent document DE 695 06 174 T2. The system has an infrared camera generating an infrared image, which shows the heat radiation of a portion of the environment of the vehicle emanating from persons and objects. The infrared image is projected by way of a head-up display onto the lower part of the windshield, and is thereby made visible to the driver.
The assignment of persons and objects in the infrared image to the traffic situation situated in front of the driver and visually detectable through the windshield is to be made by the driver. This is difficult and, often even impossible, particularly in darkness, for which the known system is intended, because the infrared image shows persons and objects which the driver cannot recognize visually. A more exact position determination of the persons and objects recognizable only in the infrared image is therefore not possible on a regular basis and dangerously diverts the driver's attention.
It is a particular object of the invention to indicate an improved method and a system for visualizing the environment of a vehicle, particularly an improved night vision system.
The object of the invention is achieved by providing a method of visualizing the environment of a vehicle, particularly in darkness, the method including: providing a visual image or its digital data of the environment, preferably a colored visual image, the visual image showing the visible objects; and providing an infrared image or its digital data of the environment, the infrared image showing the infrared radiation emanating from the visible and/or other objects. The system is characterized by a visual camera, preferably a color-sensitive visual camera, an infrared camera, a first normalizing device for normalizing the preferably colored visual image of the cutout of the environment of the vehicle provided by the visual camera, a second normalizing device for normalizing the infrared image of the cutout of the environment of the vehicle provided by the infrared camera, an aligning device for generating largely isochronous and same-location image pairs from visual images and infrared images, and a fusion or superposition device, which superposes the largely isochronous and same-location image pairs by pixels or areas and/or forms average values. Advantageous embodiments of the invention are described and claimed herein.
A first aspect of the invention consists of the display or visualization of a combination or superposition of a visual image, in the following called visual image, and of an infrared image, in the following called infrared image, of the largely identical cutout of the environment of a vehicle. As a result of this superposition or fusion of the images of different spectral ranges of the same scene, the above-described disadvantages of the state of the art are largely eliminated.
A second aspect consists of weighting the two spectral fractions (visual fraction and infrared fraction) during the fusion of the respective visual image and of the infrared image as a function of the actual driving situation relative to one another. In the first extreme case of the weighting, the image created by fusion largely has only the visual information—as a rule a superposition of visual and infrared information—, and in the second extreme case of the weighting, the image created by fusion largely has only the infrared information.
If, for example, on a country road, a vehicle, which is driving ahead, is detected by a ranging system, such as a known radar-supported automatic cruise control (ACC) system, it is provided in an embodiment of the invention that this vehicle be highlighted during the fusion of the two spectral fractions in contrast to the driving environment, so that it is more clearly visible in the fusion image in comparison with the environment. This can take place, for example, in that the weighting of the two spectral fractions for the relevant range is changed in comparison to the driving environment. In this example, here, it is that image cutout of the fusion image which shows the vehicle driving directly ahead. During the day the visual information, for example, can be weighted higher than the infrared information and can, therefore, be indicated more clearly in the fusion image, while the weighting takes place exactly the other way around during darkness.
When the vehicle is driving in a well-lit area, it is provided in another embodiment of the invention, as an alternative or in addition, that the visual fraction or the visual information be weighted higher than the infrared fraction or the infrared information. On a dark country road and/or on a turnpike, as an alternative or in addition, the weighting is carried out the other way around in comparison to the above-mentioned weighting, so that the infrared information or the infrared fraction appears more clearly than the visual fraction in the fusion image.
As an alternative or in addition, it is provided in another embodiment of the invention that, on a dark country road and/or turnpike, the infrared information of more distant objects is highlighted by their higher weighting during the fusion—in contrast to objects less distant from the vehicle. If necessary, the fusion image offered to the driver in a display may have a higher fraction of visual information in the close range and a higher fraction of infrared information in the distant range.
In rain, fog or on a wet road, the usability of the infrared channel or of the infrared information is limited compared to normal conditions. In order to counteract a deterioration of the fusion image as a result of infrared information, which at such weather conditions may at best be usable to a limited extent, it is provided, as an alternative or in addition, that the visual fraction be increased with respect to the infrared fraction; that is, the weighting of the two spectral fractions in the fusion image is changed correspondingly in the direction of visual information.
For detecting the above-mentioned weather conditions, particularly a rain sensor or a fog sensor can be provided on the vehicle. Likewise, it can be analyzed which vehicle lights are switched on, as particularly the fog lights. However, this information can also be supplied by a dynamic navigation system provided in the vehicle, into which the dynamic traffic or weather conditions for the current location of the vehicle, or particularly also in advance for the planned driving route, are transmitted. This transmission can take place by way of a cellular network or by a vehicle-to-vehicle communication.
As an alternative or in addition, it is provided in an embodiment of the invention that the driver can manually adjust the weighting of the two spectral fractions with respect to one another in the fusion image and, thereby, can optimally adapt them to his needs and special circumstances. A continuous transition of the representation from visual information to infrared information and vice-versa is provided according to an embodiment of the invention. This makes it possible for the driver to use the relationship between visual and infrared information and the influence on the fusion image of the driving environment by the displacement between the spectral fractions more easily and faster in a beneficial manner in the sense of a more rapid and more reliable detection of the traffic situation.
As an alternative or in addition, in a further embodiment, a plurality of fusion images is offered to the driver for the selection of a fusion image which appears suitable to him, the fusion images offered for the selection differing in their relationship of the fraction of visual information to infrared information. In particular, a different weighting of the visual fraction with respect to the infrared fraction may have taken place during the fusion.
In a concrete further development of the night vision system according to the invention, the visual camera or the first sensor or their lens system has a first optical axis, and the infrared camera or the second optical sensor or their lens system has a second optical axis, which are spatially offset with respect to one another. The cameras or sensors, therefore, at least partially detect different cutouts of the environment of the vehicle; that is, a first and a second cutout. In addition, these cutouts are a function of the distance. It is understood that, according to the invention, more than two infrared cameras or infrared sensors, whose sensitivity covers different wavelengths, can be provided, and the digital images of the vehicle environment supplied by them can be superposed or fused.
In order to obtain a fusion image for presentation to the driver which is largely free of distortions, it is provided in an embodiment of the invention that the provided first cutout and the provided second cutout are completely or partially superposed or fused by using a superposition or fusion device with respect to pixels and/or areas. For providing the fusion image, the first cutout and/or the second cutout and/or the fusion image or their direct or processed digital data are adapted by using at least one adaptation parameter.
This or these distance-dependent parameters are, preferably, determined during the calibration of the camera or sensor system or night vision system according to the invention for at least two distance ranges or spacing ranges between the cameras or sensors and a calibrating device. The adaptation parameter(s) are adjusted such that the fusion image of objects in the corresponding distance range created during the superposition or fusion of the images is largely without distortions, for example, particularly free of ghost images or double images. According to the invention, the adaptation parameter(s) are, particularly, at least one recording or transformation parameter. A similar parameter is known, for example, from the recording of a digital image processing system and superpositions of two images thereby implemented. The driving-situation-dependent or distance-dependent adaptation parameter(s) are, preferably, stored in a data memory in the vehicle during the calibration.
In a preferred embodiment of the invention, a first distance range corresponds to a driving situation typical of city driving, as particularly a distance range of approximately 15 to 75 m.
As an alternative or in addition, a second distance range corresponds to a driving situation typical of driving on a country road, as particularly a distance range of approximately 30 to 150 m.
As an alternative or in addition, a third distance range corresponds to a driving situation typical of driving on a turnpike, as particularly a distance range of approximately 50 to 250 m.
It is understood that, as an alternative or in addition to the above-mentioned distance-dependent or distance-range-dependent adaptation parameters, other driving-situation-dependent adaptation parameters can also be determined, particularly with the above-mentioned objective. These may, for example, be suitable adaptation parameters for drives when the sun is in a low position, in fog, at dusk, or in darkness.
Another aspect of the invention consists of automatically determining the current driving situation of the vehicle and providing the adaptation parameter(s) corresponding to the current driving situation or environmental situation, particularly obtained by calibration and stored in the vehicle, to be used by the aligning device according to the invention or by the superposition or fusion device. The superposition or fusion device carries out a pixel-type and/or area-type superposition or fusion of the visual image present in the form of digital data and of the infrared image, in which case one or more distance-dependant adaptation parameters influence the infrared image and/or the visual image and/or the fusion image, such that, preferably, a fusion image of the environment is provided to the driver which is largely free of distortions and/or ghost images.
As an alternative or in addition to the above-mentioned automatic determination, it is provided in an embodiment of the invention that the driver selects adaptation parameters, which appear suitable to him, and in particular, are obtained by calibration and are stored in the vehicle, for use by the aligning device according to the invention or by the superposition or fusion device. Thus, for example, the adaptation parameter(s) for a driving situation typical of city driving, typical of country road driving, typical of turnpike driving and/or, if required, for additional driving situations, can be offered to the driver for selection, for example, in the form of a manual selection possibility or by voice input. This enables the driver, even without a vehicle navigation system situated in the vehicle, to himself create a fusion image largely free of distortions or double images. The driver is additionally enabled, if necessary, to override an automatic selection. Likewise, it can become possible for the driver to select one or more adaptation parameter(s), which display to him the closer environment of his vehicle without distortions in the fusion image, for example, distances of up to 20 m from his vehicle. This selection can be made by the driver, for example, when he approaches his garage in darkness and wants to find out by means of the infrared information in the fusion image whether anyone is lying in wait for him behind a bush.
In a preferred embodiment of the invention, the current position of the vehicle is determined by a vehicle navigation system, particularly a satellite navigation system. By using the position information, the navigation system situated in the vehicle automatically determines, by a comparison with digital map data, the corresponding road category or driving situation, such as particularly a city street, a country road or a turnpike. Currently, such vehicle navigation systems already exist in numerous vehicles for the purpose of routing, etc. and can be utilized without any high expenditures for the above-mentioned automatic driving-situation-dependent and environment-dependent optimization of the fused images of the environment of the vehicle.
As an alternative or in addition, it is provided in another embodiment of the invention that the driving situation is determined by using at least one driving-dynamic quantity, such as particularly the vehicle speed and/or the operation of the low beam or high beam and/or the vehicle acceleration and/or brightness and/or fog. In today's vehicles, such information can completely or partially be provided through the existing electronics without requiring greater expenditures.
In a preferred embodiment of the invention, a colored visual image is fused or combined with the infrared image or superposed on the latter. In contrast to a black-and-while visual image, a fused image is created which contains colored information of the visual image, such as red brake lights of a vehicle driving ahead, a red traffic light, a colored traffic sign, or the like. The colored information in the fused image facilitates a fast orientation and detection of the traffic situation shown in the fused image for the driver of a correspondingly equipped vehicle. In addition, the color information reduces the risk that colored warning signs (red traffic lights, etc.) are overlooked.
Summarizing, the images of the environment or scene for different spectral ranges are, in each case, freed of disturbing influences, such as distortions of the infrared or visual image, particularly as a result of the used lens system, etc. With respect to the software, this preferably takes place by known measures for processing digitized images. The images or their digital image data largely freed of disturbing influences preferably are aligned with respect to one another or made congruent by digital image processing, so that largely isochronous and same-location image pairs of the infrared and visual image or their digital data are present. According to the invention, this takes place by using at least one distance-dependent and/or driving-situation-dependent adaptation parameter for obtaining a distortion-free fusion image. The software and hardware used for the digital image processing preferably allows a displacement, a rotation, and a scaling of the images relative to one another. As a result of this processing, the subsequent hardware expenditures for the superposition or fusion of the images from the different spectral ranges can be minimized in a cost-effective manner—despite a largely real-time processing of isochronous and same-location image pairs.
According to a preferred embodiment of the invention, the infrared images and the visual images are generated with a respectively identical image repetition rate, preferably by one or two cameras or sensors for these spectral ranges. As a result, isochronous image pairs of the infrared image and the visual image can be generated in a particularly simple manner, which considerably reduces the software and hardware expenditures for the subsequent superposition or fusion of the two images according to the invention—largely in real time. The processing speed of the image pairs increases; the costs for the semiconductor memory for the intermediate storage of the images are minimized.
In a preferred embodiment of the invention, the infrared image shows the infrared radiation or heat radiation emanating from persons and objects, which radiation is in a wavelength range from approximately 8 to 14 μm. Preferably, an IR camera or an IR sensor is used, which is sensitive in the range from approximately 8 to 10 μm. This advantageously eliminates an infrared radiator or other such illumination (typical wavelength range of approximately 800 nm to 2.5 μm) in the vehicle environment. A mutual blinding of oncoming vehicles, which in a known manner are each equipped with an infrared illumination, does not occur. Likewise, the infrared image according to the invention is not limited to the range of an infrared illumination.
Other objects, advantages and novel features of the present invention will become apparent from the following detailed description of the invention when considered in conjunction with the accompanying drawings.